-
Notifications
You must be signed in to change notification settings - Fork 403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[5.2] Build occurrences index for Merlin #10422
Conversation
For a first release, I would consider the following strategy to select whether this feature is enabled:
(This is similar to how we set up That way:
|
otherlibs/stdune/src/stdune.ml
Outdated
@@ -1,5 +1,4 @@ | |||
[@@@alert unstable "The API of this library is not stable and may change without notice."] | |||
[@@@alert "-unstable"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just so we don't forget: you can work around that by passing -alert -unstable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not familiar with alerts, I tried to add -alert
but it still fails:
File "otherlibs/stdune/src/stdune.ml", line 2, characters 0-29:
2 | [@@@alert "-alert -unstable"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error (warning 47 [attribute-payload]): illegal payload for attribute 'alert'.
Ill-formed list of alert settings
File "otherlibs/stdune/src/stdune.ml", line 1, characters 4-9:
1 | [@@@alert unstable "The API of this library is not stable and may change without notice."]
^^^^^
Error (warning 53 [misplaced-attribute]): the "alert" attribute cannot appear in this context
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
taking a step back: why did you make that change? in other words, what breaks when you leave these attributes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get this error when I build with 5.2~beta2
:
File "otherlibs/stdune/src/stdune.ml", line 1, characters 4-9:
1 | [@@@alert unstable "The API of this library is not stable and may change without notice."]
^^^^^
Error (warning 53 [misplaced-attribute]): the "alert" attribute cannot appear in this context
I found that related change in the compiler: ocaml/ocaml#12451
@ccasin I am not sure if our usage of [@@@alert deprecated
here is indeed wrong or if the warning is triggering unexpectedly?
I wonder whether we should attach |
@rgrinberg removing the aggregation step does have an impact on cold build time, which is probably more noticeable on rebuilds.
But now I realize that my measurement are wrong: the If I compare with
But even then, there is some additional building happening (in vendored dirs notably). I find this a bit surprising since the dependencies of Additionally, is there some way to list the targets built for some given aliases ? And to have Dune output time spent running commands (ideally total time per tool) ? In any case, the more I think about it the more I am convinced that the "don't merge" strategy is the best. I will update the merlin config and see how it goes. |
As discussed, the performance penalty currently is too large to build this by default. A separate alias sounds like a good start to me. When this feature becomes fast enough for general use, do you think we will still need the ocaml-index alias? Also, could you comment on the stability of this feature for the users? When this feature is shipped, is it recommended that everyone start using it right away? Or is it more for wider experimentation? |
Thanks for your review @rgrinberg. I was focused on experimenting with performance improvement but I will answer your comments soon. We spent some time with @emillon to make the indexer faster and we did found some important optimizations. However indexing only the libraries/executables which are part of In short, I don't think we can afford to enable the feature by default right now. I propose to add two additional targets:
Eventually, our best option is probably to have LSP ask dune via RPC to update the index when needed.
Results are actually better on a codebase such as Irmin where there is no single very large library such as
There is a tension between: more granularity for smaller incremental rebuild VS less granularity that allow for greater use of cache which speeds up cold builds but slow down incremental rebuilds. We could probably adapt the indexer to cache on the disk results for each cmt so that we could get the best of both worlds, but I don't think the benefit would be big enough to make it a priority. |
Yes, there's the |
Could you stick to |
6da9b18
to
3173f97
Compare
I rebased the PR and made a few more changes like adding documentation and removing I think the PR is now ready for another round of review. @rgrinberg I still wonder about the hidden transitive deps issue. I think now that the compiler supports the |
I think it's correct though not ideal. I can't think of a better way |
Right, I reworked the update of Merlin's configuration to use this set and also rely on the new I moved these changes to a separated PR since they are a bit orthogonal: #10535 |
List.rev_append cmts acc) | ||
in | ||
Command.run_dyn_prog | ||
~dir:context_dir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you not run ocaml-index from dir
? We only run compilation commands from context_dir for good error messages. Does ocaml-index output error messages that refer to source code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ocaml-index
relies on the load-path stored in the cmt
files. These paths are relative to the directory in which the compiler was called.
If you would prefer the indexer to run in dir
we can have dune pass the complete load-path via the CLI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, that's fine to leave it this way. It makes sense to leave a comment about this though.
Signed-off-by: Ulysse Gérard <[email protected]>
We use a mock binary in tests Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]> Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
They should all be relative to the root of the build directory Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
Signed-off-by: Ulysse Gérard <[email protected]>
- `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. Signed-off-by: Ulysse Gérard <[email protected]> Signed-off-by: Rudi Grinberg <[email protected]> Co-authored-by: Rudi Grinberg <[email protected]>
CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro)
CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro) - virtual libraries: fix an issue where linking an executable involving several virtual libries would cause an error. (ocaml/dune#10581, fixes ocaml/dune#10460, @rgrinberg)
CHANGES: ### Added - allow libraries with the same `(name ..)` in projects as long as they don't conflict during resolution (via `enabled_if`). (ocaml/dune#10307, @anmonteiro, @jchavarri) - `dune describe pp` now finds the exact module and the stanza it belongs to, instead of guessing the name of the preprocessed file. (ocaml/dune#10321, @anmonteiro) - Print the result of `dune describe pp` with the respective dialect printer. (ocaml/dune#10322, @anmonteiro) - Add new flag `--context` to `dune ocaml-merlin`, which allows to select a Dune context when requesting Merlin config. Add `dune describe contexts` subcommand. Introduce a field `generate_merlin_rules` for contexts declared in the workspace, that allows to optionally produce Merlin rules for other contexts besides the one selected for Merlin (ocaml/dune#10324, @jchavarri) - melange: add include paths for private library `.cmj` files during JS emission. (ocaml/dune#10416, @anmonteiro) - `dune ocaml-merlin`: communicate additional directives `SOURCE_ROOT`, `UNIT_NAME` (the actual name with wrapping) and `INDEX` with the paths to the index(es). (ocaml/dune#10422, @voodoos) - Add a new alias `@ocaml-index` that uses the `ocaml-index` binary to generate indexes that can be read by tools such as Merlin to provide project-wide references search. (ocaml/dune#10422, @voodoos) - merlin: add optional `(merlin_reader CMD)` construct to `(dialect)` stanza to configure a merlin reader (ocaml/dune#8567, @andreypopp) ### Changed - melange: treat private libraries with `(package ..)` as public libraries, fixing an issue where `import` paths were wrongly emitted. (ocaml/dune#10415, @anmonteiro) - install `.glob` files for Coq theories too (ocaml/dune#10602, @ejgallego) ### Fixed - Don't try to document non-existent libraries in doc-new target (ocaml/dune#10319, fixes ocaml/dune#10056, @jonludlam) - Make `dune-site`'s `load_all` function look for `META` files so that it doesn't fail on empty directories in the plugin directory (ocaml/dune#10458, fixes ocaml/dune#10457, @shym) - Fix incorrect warning for libraries defined inside non-existant directories using `(subdir ..)` and used by executables using `dune-build-info` (ocaml/dune#10525, @rgrinberg) - Don't try to take build lock when running `coq top --no-build` (ocaml/dune#10547, fixes ocaml/dune#7671, @lzy0505) - Make sure to truncate dune's lock file after locking and unlocking so that users cannot observe incorrect pid's (ocaml/dune#10575, @rgrinberg) - mdx: link mdx binary with `byte_complete`. This fixes `(libraries)` with foreign archives on Linux. (ocaml/dune#10586, fixes ocaml/dune#10582, @anmonteiro) - virtual libraries: fix an issue where linking an executable involving several virtual libries would cause an error. (ocaml/dune#10581, fixes ocaml/dune#10460, @rgrinberg)
This PR adds new rules that drive the
ocaml-index
tool to aggregate a workspace index. The rules are a bit simpler as the ones in #6762 since now part of the indexation will is done by the compiler in 5.2.Building the index involves three steps:
-bin-annot-occurrences
flag to have the compiler store partial indexes in thecmt
files.ocaml-index
tool to make an index for each stanza (actually each compilation-context).ocaml-index
tool to aggregate every stanza's index into a global project index.I ran simple benchmarks on Dune itself. These are cold builds ran after
./dune.exe clean
:./dune.exe build @check
(without-bin-annot-occurrences
)./dune.exe build @check
./dune.exe build @check @ocaml-index
-bin-annon-occurrences
flag when writingcmt
files, so the best option is probably to always use that flag to simplify the rules.There are several moving parts that I would like to discuss in the next dune-dev meeting:
ocaml-index
tool must be installed-bin-annot-occurrences
flag requires OCaml 5.2dune-workspace
-level option that I remove in cd34ae1 following a discussion with @emillon@ocaml-index
alias but remove it from@default
so that users have to explicitly use it.Other issues to consider:
-bin-annot-occurrences
flag